Video display system, video display method, and video display program

ABSTRACT

An video display system having room in a generation time of a next frame to be generated on the basis of a gaze direction of the user is provided. A video display system includes a wearable device including a reception unit that receive a video, a display unit that displays the video received by the reception unit, an irradiation unit that irradiates eyes of a user with near infrared light, and an imaging unit that images the eyes of the user viewing the video displayed on the display unit, on the basis of the near infrared light, the user wearing the wearable device and viewing the video; a gaze detection unit that detects a gaze point of the user on the basis of a captured image captured by the imaging unit; and a video generation unit that generates a video to be displayed on the wearable device on the basis of the gaze point detected by the gaze detection unit, and the wearable device includes a control unit that instructs an imaging start timing to the imaging unit so that the imaging of the imaging unit can be executed at a timing at which it is estimated that the user is viewing the frame each time each frame of the video to be displayed on the display unit is displayed.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a video display system, a video displaymethod, and a video display program for generating and displaying avideo on the basis of a gaze point of a user.

Description of Related Art

In the related art, development of head mounted displays, wearableglasses, and the like as devices that are mounted on the head of a userto present videos has progressed, and some of these detect the gaze ofthe user. Japanese Unexamined Patent Application Publication No.2015-90569 discloses an information processing device capable ofdetecting a gaze direction of a user, and discloses acquiring an imageof the eyes of the user at every predetermined timing.

SUMMARY OF THE INVENTION

Meanwhile, such a detection of the gaze may be used, for example, increation of video data of the next frame in a video. More specifically,for example, when a user is viewing a 360-degree video, video data ofthe next frame based on the gaze of the user is created and displayed.In this case, the gaze is specified using a captured image obtained byimaging the eyes of the user viewing the video, but in imaging at everypredetermined timing, there is a problem in that the fact that a frameof the captured image is actually viewed by the user is not guaranteed.Since video data is prepared through a process of imaging the eyes ofthe user, analyzing a captured image obtained by the imaging, andspecifying a gaze point in preparing the next frame, there is also aproblem in that a usable time for preparing the video data is short.

The present invention has been made in view of the above problems, andan object of the present invention is to provide a video display systemthat can obtain a captured image of the eyes of a user viewing eachframe more accurately and can secure a longer period of time forpreparation of video data to be viewed by the user than in the relatedart.

In order to solve the above problem, a video display system according toan aspect of the present invention includes a wearable device includinga reception unit that receive a video, a display unit that displays thevideo received by the reception unit, an irradiation unit thatirradiates eyes of a user with near infrared light, and an imaging unitthat images the eyes of the user viewing the video displayed on thedisplay unit, on the basis of the near infrared light, the user wearingthe wearable device and viewing the video; a gaze detection unit thatdetects a gaze point of the user on the basis of a captured imagecaptured by the imaging unit; and a video generation unit that generatesa video to be displayed on the wearable device on the basis of the gazepoint detected by the gaze detection unit, wherein the wearable deviceincludes a control unit that instructs an imaging start timing to theimaging unit so that the imaging of the imaging unit can be executed ata timing at which it is estimated that the user is viewing the frameeach time each frame of the video to be displayed on the display unit isdisplayed.

In order to solve the above problem, a video display method according toan aspect of the present invention includes a reception step ofreceiving a video; a display step of displaying the video received inthe reception step; an irradiation step of irradiating eyes of a userwith near infrared light; a control step of instructing an imaging starttiming so that imaging can be executed at a timing at which it isestimated that the user is viewing the frame each time the frame of thevideo to be displayed in the display step is displayed; an imaging stepof imaging the eyes of the user viewing the video displayed in thedisplay step, on the basis of the near infrared light according to theinstructed start timing; a gaze detection step of detecting a gaze pointof the user on the basis of a captured image captured in the imagingstep; and a video generation step of generating a video to be displayedon the basis of the gaze point detected in the gaze detection step.

In order to solve the above-mentioned problems, a video display programaccording to one aspect of the present invention causes a computer toexecute a reception function of receiving a video; a display function ofdisplaying the video received using the reception function; anirradiation function of irradiating eyes of a user with near infraredlight; a control function of instructing an imaging start timing so thatimaging can be executed at a timing at which it is estimated that theuser is viewing the frame each time the frame of the video to bedisplayed using the display function is displayed; an imaging functionof imaging the eyes of the user viewing the video displayed using thedisplay function, on the basis of the near infrared light according tothe instructed start timing; a gaze detection function of detecting agaze point of the user on the basis of a captured image captured usingthe imaging function; and a video generation function of generating avideo to be displayed on the basis of the gaze point detected using thegaze detection step.

Further, in the video display system, the reception unit may convertvideo data of the received video into a format for displaying the videoon the display unit, the display unit may output a synchronizationsignal indicating start of display of the video data converted by thereception unit, and the control unit may output, to the imaging unit, aninstruction signal for instructing the start timing to the imaging uniton the basis of the synchronization signal.

In the video display system, the control unit may output the instructionsignal according to a period of time from start of display of the frameon the display unit to viewing by the user.

Further, in the video display system, the control unit may furthercontrol a timing at which the irradiation unit irradiates the eyes ofthe user with the near-infrared light.

The video display system according to an aspect of the present inventioncan image the eyes of the user viewing the video more accurately bycontrolling the timing for imaging the eyes of the user. In addition,the video display system can give room for preparation of video data tobe viewed next by advancing the start of the imaging timing relative tothat in the related art through imaging timing control.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a block diagram illustrating a configuration example of avideo display system.

FIG. 2 is an external view illustrating a state in which a head mounteddisplay is worn by a user.

FIG. 3 is a perspective view schematically illustrating an overview ofan image display system of a head mounted display.

FIG. 4 is a diagram schematically illustrating an optical configurationof an image display system of a head mounted display.

FIG. 5 is a schematic diagram illustrating calibration for detection ofa gaze direction.

FIG. 6 is a schematic diagram illustrating position coordinates of acornea of a user.

FIG. 7 is a flowchart illustrating an operation example of a videodisplay system.

FIG. 8 is a timing chart of a case in which timing control of imagingtiming is not performed.

FIG. 9 is a timing chart of a case in which timing control of imagingtiming is performed.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, embodiments of a video display system according to thepresent invention will be described with reference to the drawings.

Embodiment

As illustrated in FIG. 1, the video display system according to thepresent invention includes a wearable device 100 including a receptionunit 110 that receives a video, a display unit 102 that displays thevideo received by the reception unit, an irradiation unit 135 thatirradiates eyes of a user with near infrared light, and an imaging unit140 that images the eyes of the user viewing the video displayed on thedisplay unit 102, on the basis of the near infrared light, the userwearing the wearable device and viewing the video; a gaze detectiondevice 200 that detects a gaze point of the user on the basis of acaptured image captured by the imaging unit; and a video generation unit250 that generates a video to be displayed on the wearable device 100 onthe basis of the gaze point detected by a gaze detection unit 220,wherein the wearable device 100 includes a control unit 120 thatinstructs an imaging start timing to the imaging unit 140 so that theimaging of the imaging unit 140 can be executed at a timing at which itis estimated that the user is viewing the frame each time each frame ofthe video to be displayed on the display unit 102 is displayed.

That is, the video display system 1 can perform imaging at a timing atwhich it is estimated that the user is actually viewing the video(frames) displayed on the display unit 102 by controlling the imagingtiming of the imaging unit 140, and then generate a video (frames) to beviewed next by the user with some room. Hereinafter, the video displaysystem 1 will be described in detail.

FIG. 2 is a diagram schematically illustrating an overview of the videodisplay system 1 according to the embodiment. The video display system 1according to the embodiment includes a head mounted display 100illustrated as an example of a wearable device 100 and a gaze detectiondevice 200. Hereinafter, the wearable device 100 is described as a headmounted display 100. As illustrated in FIG. 2, the head mounted display100 is mounted on the head of a user 300 for use.

The gaze detection device 200 detects a gaze direction of at least oneof the right and left eyes of the user wearing the head mounted display100, and specifies a focal point of the user, that is, a gaze point ofthe user in a three-dimensional image displayed on the head mounteddisplay. Further, the gaze detection device 200 also functions as avideo generation device that generates videos displayed by the headmounted display 100. For example, the gaze detection device 200 is adevice capable of reproducing videos of stationary game machines,portable game machines, PCs, tablets, smartphones, phablets, videoplayers, TVs, or the like, but the present invention is not limitedthereto. The gaze detection device 200 is connected wirelessly or by awire to the head mounted display 100. In the example illustrated in FIG.2, the gaze detection device 200 is connected to the head mounteddisplay 100 by a cable (for example, a USB cable), but may be wirelesslyconnected thereto. In the case of the wireless connection, the wirelessconnection to be executed between the gaze detection device 200 and thehead mounted display 100 can be realized using a known wirelesscommunication technique such as Wi-Fi (registered trademark) orBluetooth (registered trademark). For example, transfer of videosbetween the head mounted display 100 and the gaze detection device 200is executed according to a standard such as Miracast (registeredtrademark), WiGig (registered trademark), or WHDI (registeredtrademark), and the present invention is not limited thereto.

It should be noted that FIG. 2 illustrates an example in a case in whichthe head mounted display 100 and the gaze detection device 200 aredifferent devices. However, the gaze detection device 200 may beembedded in the head mounted display 100.

The head mounted display 100 includes a housing 150, a fitting harness160, and headphones 170. The housing 150 houses an image display system,such as an image display element, for presenting videos to the user 300,and a wireless transfer module (not illustrated) such as a Wi-Fi moduleor a Bluetooth (registered trademark) module. The fitting harness 160 isused to mount the head mounted display 100 on the head of the user 300.The fitting harness 160 may be realized by, for example, a belt or anelastic band. When the user 300 wears the head mounted display 100 usingthe fitting harness 160, the housing 150 is arranged at a position wherethe eyes of the user 300 are covered. Thus, if the user 300 wears thehead mounted display 100, a field of view of the user 300 is covered bythe housing 150.

The headphones 170 output audio for the video that is reproduced by thegaze detection device 200. The headphones 170 may not be fixed to thehead mounted display 100. Even when the user 300 wears the head mounteddisplay 100 using the fitting harness 160, the user 300 may freelyattach or detach the headphones 170.

FIG. 3 is a perspective diagram illustrating an overview of the imagedisplay system 130 of the head mounted display 100 according to theembodiment. Specifically, FIG. 3 illustrates a region of the housing 150according to an embodiment that faces corneas 302 of the user 300 whenthe user 300 wears the head mounted display 100.

As illustrated in FIG. 3, a convex lens 114 a for the left eye isarranged at a position facing the cornea 302 a of the left eye of theuser 300 when the user 300 wears the head mounted display 100.Similarly, a convex lens 114 b for a right eye is arranged at a positionfacing the cornea 302 b of the right eye of the user 300 when the user300 wears the head mounted display 100. The convex lens 114 a for theleft eye and the convex lens 114 b for the right eye are gripped by alens holder 152 a for the left eye and a lens holder 152 b for the righteye, respectively.

Hereinafter, in this specification, the convex lens 114 a for the lefteye and the convex lens 114 b for the right eye are simply referred toas a “convex lens 114” unless the two lenses are particularlydistinguished. Similarly, the cornea 302 a of the left eye of the user300 and the cornea 302 b of the right eye of the user 300 are simplyreferred to as a “cornea 302” unless the corneas are particularlydistinguished. The lens holder 152 a for the left eye and the lensholder 152 b for the right eye are referred to as a “lens holder 152”unless the holders are particularly distinguished.

A plurality of infrared light sources 103 are included in the lensholders 152. For the purpose of brevity, in FIG. 3, the infrared lightsources that irradiate the cornea 302 a of the left eye of the user 300with infrared light are collectively referred to as infrared lightsources 103 a, and the infrared light sources that irradiate the cornea302 b of the right eye of the user 300 with infrared light arecollectively referred to as infrared light sources 103 b. Hereinafter,the infrared light sources 103 a and the infrared light sources 103 bare referred to as “infrared light sources 103” unless the infraredlight sources 103 a and the infrared light sources 103 b areparticularly distinguished. In the example illustrated in FIG. 3, sixinfrared light sources 103 a are included in the lens holder 152 a forthe left eye. Similarly, six infrared light sources 103 b are includedin the lens holder 152 b for the right eye. Thus, the infrared lightsources 103 are not directly arranged in the convex lenses 114, but arearranged in the lens holders 152 that grip the convex lenses 114, makingthe attachment of the infrared light sources 103 easier. This is becausemachining for attaching the infrared light sources 103 is easier thanfor the convex lenses 114 that are made of glass or the like since thelens holders 152 are typically made of a resin or the like.

As described above, the lens holders 152 are members that grip theconvex lenses 114. Therefore, the infrared light sources 103 included inthe lens holders 152 are arranged around the convex lenses 114. Althoughthere are six infrared light sources 103 that irradiate each eye withinfrared light herein, the number of the infrared light sources 103 isnot limited thereto. There may be at least one light source 103 for eacheye, and two or more light sources 103 are desirable.

FIG. 4 is a schematic diagram of an optical configuration of the imagedisplay system 130 contained in the housing 150 according to theembodiment, and is a diagram illustrating a case in which the housing150 illustrated in FIG. 3 is viewed from a side surface on the left eyeside. The image display system 130 includes infrared light sources 103,an image display element 108, a hot mirror 112, the convex lenses 114, acamera 116, and an output unit 180.

The infrared light sources 103 are light sources capable of emittinglight in a near-infrared wavelength region (700 nm to 2500 nm range).Near-infrared light is generally light in a wavelength region ofnon-visible light that cannot be observed by the naked eye of the user300.

The image display element 108 displays an image to be presented to theuser 300. An image to be displayed by the image display element 108 isgenerated by the video generation unit 232 in the gaze detection device200. The video generation unit 232 will be described below. The imagedisplay element 108 can be realized by using, for example, a knownliquid crystal display (LCD) or an organic electro luminescence display(EL display).

The hot mirror 112 is arranged between the image display element 108 andthe cornea 302 of the user 300 when the user 300 wears the head mounteddisplay 100. The hot mirror 112 has a property of transmitting visiblelight created by the image display element 108, but reflectingnear-infrared light.

The convex lenses 114 are arranged on the opposite side of the imagedisplay element 108 with respect to the hot mirror 112. In other words,the convex lenses 114 are arranged between the hot mirror 112 and thecornea 302 of the user 300 when the user 300 wears the head mounteddisplay 100. That is, the convex lenses 114 are arranged at positionsfacing the corneas 302 of the user 300 when the user 300 wears the headmounted display 100.

The convex lenses 114 condense image display light that is transmittedthrough the hot mirror 112. Thus, the convex lenses 114 function asimage magnifiers that enlarge an image created by the image displayelement 108 and present the image to the user 300. Although only one ofeach convex lens 114 is illustrated in FIG. 3 for convenience ofdescription, the convex lenses 114 may be lens groups configured bycombining various lenses or may be a plano-convex lens in which onesurface has curvature and the other surface is flat.

A plurality of infrared light sources 103 are arranged around the convexlens 114. The infrared light sources 103 emit infrared light toward thecornea 302 of the user 300.

Although not illustrated in the figure, the image display system 130 ofthe head mounted display 100 according to the embodiment includes twoimage display elements 108, and can independently generate an image tobe presented to the right eye of the user 300 and an image to bepresented to the left eye of the user. Accordingly, the head mounteddisplay 100 according to the embodiment may present a parallax image forthe right eye and a parallax image for the left eye to the right andleft eyes of the user 300. Thereby, the head mounted display 100according to the embodiment can present a stereoscopic video that has afeeling of depth for the user 300.

As described above, the hot mirror 112 transmits visible light butreflects near-infrared light. Thus, the image light emitted by the imagedisplay element 108 is transmitted through the hot mirror 112, andreaches the cornea 302 of the user 300.

The infrared light reaching the cornea 302 of the user 300 is reflectedby the cornea 302 of the user 300 and is directed to the convex lens 114again. This infrared light is transmitted through the convex lens 114and is reflected by the hot mirror 112. The camera 116 includes a filterthat blocks visible light and images the near-infrared light reflectedby the hot mirror 112. That is, the camera 116 is a near-infrared camerawhich images the near-infrared light emitted from the infrared lightsources 103 and reflected by the cornea of the eye of the user 300.

Although not illustrated in the figure, the image display system 130 ofthe head mounted display 100 according to the embodiment includes twocameras 116, that is, a left-eye imaging camera 116 a that captures animage including the infrared light reflected by the right eye and aright-eye imaging camera 116 b that captures an image including theinfrared light reflected by the left eye. Thereby, images for detectinggaze directions of both the right eye and the left eye of the user 300can be acquired.

The output unit 180 outputs the image captured by the camera 116 to thegaze detection device 200 that detects the gaze direction of the user300. Specifically, the output unit 180 transmits the image captured bythe camera 116 to the gaze detection device 200. Although the gazedetection unit 220 will be described below in detail, the gaze directionunit is realized by a gaze detection program executed by a centralprocessing unit (CPU) of the gaze detection device 200. When the headmounted display 100 includes computational resources such as a CPU or amemory, the CPU of the head mounted display 100 may execute the programthat realizes the gaze direction detection unit.

As will be described below in detail, bright spots caused bynear-infrared light reflected by the cornea 302 of the user 300 and animage of the eyes including the cornea 302 of the user 300 observed in anear-infrared wavelength region are captured in the image captured bythe camera 116. The near-infrared light from the infrared light sourcehas some degree of directivity, but also radiates a certain degree ofdiffused light, and the image of the eyes of the user 300 is capturedwith the diffused light.

Although the configuration for presenting the image to the left eye ofthe user 300 in the image display system 130 according to the embodimenthas mainly been described above, a configuration for presenting an imageto the right eye of the user 300 is the same as above.

Referring back to FIG. 1, FIG. 1 is a block diagram illustrating adetailed configuration of the video display system 1. As illustrated inFIG. 1, the video display system 1 includes a head mounted display 100and a gaze detection device 200.

As illustrated in FIG. 1, the head mounted display 100 includes areception unit 110, a display unit 102, a control unit 120, anirradiation unit 135, an imaging unit 140, and an output unit 180. Thereception unit 110, the display unit 102, the control unit 120, theirradiation unit 135, the imaging unit 140, and the output unit 180 arerealized by different circuits, and are connected as shown in FIG. 1.

The reception unit 110 has a function of receiving video data to bedisplayed on the display unit 102. The reception unit 110 includes aninput terminal 111 and an output terminal 113. The input terminal 111receives video data as, for example, a video signal. The reception unit110 converts the received video signal into a mobile industry processorinterface (MIPI) format. The converted video signal is output from theoutput terminal 113 and transferred to the display unit 102. Thereception unit 110 is an input interface that receives video data, andthe input terminal 111 may be, for example, a USB terminal. It should benoted that although the received video signal is converted to the MIPIformat herein, this is only an example, and the signal may be convertedinto a signal in other formats such as a low voltage differentialsignaling (LVDS) signal or a baseband signal. The same apply below.Also, for the input terminal, the USB terminal is merely an example, theinput terminal may be another terminal or input mechanism, or HDMI(registered trademark) (High Definition Multimedia Interface), a displayport, a wireless communication chip, or the like may be used.

The display unit 102 is a display unit having a function of displayingthe video data received by the reception unit 110. The display unit 102receives the transferred video signal in a MIPI format at the inputterminal 104 and displays an image based on the received video signal onthe image display element 108. In addition, the display unit 102 uses avertical synchronization signal as a writing start trigger, outputs thevertical synchronization signal, and also outputs from the outputterminal 106 to the control unit 120. It should be noted that althoughthe configuration in which the vertical synchronization signal is outputfrom the output terminal 106 is illustrated herein, a configuration inwhich the vertical synchronization signal is output from thesynchronization output of the reception unit 110 to the control unit 120may be used. In addition, the display unit 102 displays the marker imageoutput from the video generation unit 250 at the designated coordinatesof the image display element 108.

The control unit 120 has a function of controlling the timing of imagingin the imaging unit 140. The control unit 120 also has a function ofcontrolling the timing of irradiation with the near-infrared light fromthe infrared light source 103 by the irradiation unit 135. The controlunit 120 includes an input terminal 121, an input terminal 122, anirradiation delay control unit 123, an imaging delay control unit 124,an output terminal 125, and an output terminal 126. The control unit 120is, for example, a microprocessor.

The input terminal 121 is a terminal for connection to the gazedetection device 200, and receives the viewing timing from the gazedetection device 200, and transfers the viewing timing to theirradiation delay control unit 123 and the imaging delay control unit124. Here, the viewing timing is information on a time when the user 300actually views content of one frame of the video displayed on thedisplay unit 102 and the gaze point is determined. The time of theviewing timing is changed (before and after) due to a difference inreaction speed by the user 300.

The input terminal 122 is a terminal for receiving the verticalsynchronization signal output from the display unit 102, and transfersthe timing for receiving the signal to the imaging delay control unit124.

The irradiation delay control unit 123 has a function of controlling anirradiation timing for irradiating the eyes of the user 300 with nearinfrared light using the left eye LED 103 a and the right eye LED 103 bof the irradiation unit 135. Specifically, the irradiation delay controlunit 123 generates an irradiation timing signal so that the eyes of theuser are irradiated with infrared light at the imaging timing on thebasis of the viewing timing transferred from the input terminal 121 andthe timing of the vertical synchronization signal transferred from theinput terminal 122, and transfers the irradiation timing signal to theoutput terminal 125. It is desirable for the irradiation timing to beslightly earlier than the imaging timing of the imaging unit 140. Itshould be noted that the irradiation timing strictly indicates a timingfor irradiating the eyes of the user with the near-infrared light at atiming for imaging the eyes of the user in the next frame of the videodata. By controlling the timing for irradiating the eyes of the userwith the near-infrared light from the infrared light source 103 (theLEDs 103 a and 103 b) and performing ON/OFF, it is possible to achievepower saving in the irradiation unit 135.

The imaging delay control unit 124 has a function of controlling thetiming at which the imaging unit 140 images the eyes of the user withthe left-eye imaging camera 116 a and the right-eye imaging camera 116b. The imaging delay control unit 124 generates the imaging timingsignal indicating the imaging timing at which the imaging unit 140starts imaging on the basis of the viewing timing transferred from theinput terminal 121 and the timing of the vertical synchronization signaltransferred from the input terminal 122, and transmits the imagingtiming signal to the output terminal 126. The imaging timing signal is asignal for instructing activation of imaging so that imaging isperformed at a timing at which the user 300 actually views one frame ofthe video displayed by the display unit 102 and the gaze point isdetermined. It should be noted that the imaging timing strictlyindicates a timing at which the eyes of the user is imaged in the nextframe of the video data.

The output terminal 125 has a function of transferring the transferredirradiation timing signal to the irradiation unit 135.

The output terminal 126 has a function of transferring the transferredimaging timing signal to the imaging unit 140.

The irradiation unit 135 has a function of irradiating the eyes of theuser with the near-infrared light. The irradiation unit 135 includes aleft eye LED (near infrared light source) 103 a and a right eye LED(near infrared light source) 103 b. The irradiation unit 135 turns onthe left eye LED 103 a and the right eye LED 103 b according to theirradiation timing signal transferred from the output terminal 125.

The imaging unit 140 has a function of imaging the eyes of the user 300on the basis of the near infrared light radiated by the irradiation unit135. The imaging unit 140 includes a left-eye imaging camera 116 a and aright-eye imaging camera 116 b. The imaging unit 140 activates theleft-eye imaging camera 116 a and the right-eye imaging camera 116 baccording to the imaging timing signal transferred from the outputterminal 126, and images the eyes of the user 300 viewing the video. Theimaging unit 140 transfers the captured image to the output unit 180. Itshould be noted that the control of the imaging timing may becontrolling a timing at which a global shutter is opened.

The output unit 180 is an interface having a function of outputting thecaptured image captured by the imaging unit 140 to the gaze detectiondevice 200. The output unit 180 includes an input terminal 182 and anoutput terminal 181. The output unit 180 transfers the captured imagereceived from the imaging unit 140 at the input terminal 182 to the gazedetection device 200 via the output terminal 181. The output terminal181 can be realized by, for example, a USB terminal.

The above is the configuration of the head mounted display 100. Next,the gaze detection device 200 will be described.

The gaze detection device 200 includes an input terminal 210, a gazedetection unit 220, a video generation unit 250, an output terminal 230,and an output terminal 240.

The input terminal 210 is a terminal for receiving an input of thecaptured image transmitted from the head mounted display 100. The inputterminal 210 can be realized by, for example, a USB terminal. The inputterminal 210 transfers the received captured image to the gaze detectionunit 220.

The gaze detection unit 220 has a function of detecting the gaze of theuser 300 on the basis of the transferred captured image. The gazedetection unit 220 can be realized by, for example, a microprocessor.The gaze detection unit 220 includes an image analysis unit 221, adetection unit 222, and an imaging control unit 223.

The image analysis unit 221 specifies a corneal position of the user 300or the like from the transferred captured image.

The detection unit 222 detects a gaze point in the image of the user 300using an analysis result of the captured image of the image analysisunit 221. The detection unit 222 transfers the detected gaze point tothe video generation unit 250.

The imaging control unit 223 specifies a time required for the user 300to actually view the displayed video and transfers the time to theoutput terminal 230 as the viewing timing. For example, the imagingcontrol unit 223 uses a default value as a predetermined time requireduntil a person generally clearly views a target or a delay estimationvalue obtained from a time interval between an image change timing and agaze point change timing to specify a time required for viewing.

Further details of the scheme of detecting the gaze of the user 300 inthe gaze detection unit 220 will be described below.

The video generation unit 250 generates video data of the next frame onthe basis of the gaze point of the user 300 transferred from thedetection unit 222. For example, the video generation unit 250 generatesvideo data of a predetermined range (for example, a display range inwhich a video can be displayed on the image display element 108 of thehead mounted display 100) around the gaze point of the user 300.Further, the video generation unit 250 may generate, for example, thevideo data with the predetermined range around the gaze point of theuser 300 being at a high resolution and the outside of the predeterminedrange being at a low resolution. The generated video data is transferredto the output terminal 240.

The output terminal 230 has a function of transferring the viewingtiming transferred from the imaging control unit 223 to the head mounteddisplay 100. The output terminal 230 can be realized by, for example, aUSB terminal.

The output terminal 240 has a function of transferring the video datatransferred from the video generation unit 250 to the head mounteddisplay 100 as a video signal. The output terminal 240 can be realizedby, for example, a USB terminal, but HDMI, a display port, and the likecan be used, similar to the input terminal. Further, the output terminal240 transfers a video data output timing to the imaging control unit223. Accordingly, the imaging control unit 223 can transfer the viewingtiming in synchronization with the video data to be output, andtherefore, the head mounted display 100 can realize imaging synchronizedwith the video data output from the gaze detection device 200.

The above is the description of the configuration of the gaze detectiondevice 200. Next, the detection of the gaze point of the user will bedescribed.

FIG. 5 is a schematic diagram illustrating calibration for detecting thegaze direction according to the embodiment. The gaze direction of theuser 300 is realized by the gaze detection unit 220 in the gazedetection device 200 analyzing the video captured by the camera 116 andoutput to the gaze detection device 200 by the output unit 180.

The video generation unit 232 generates nine points (marker images)including points Q₁ to Q₉ as illustrated in FIG. 5, and causes thepoints to be displayed by the image display element 108 of the headmounted display 100. The gaze detection device 200 causes the user 300to sequentially gaze at the points Q₁ up to Q₉. In this case, the user300 is requested to gaze at each of the points by moving his or hereyeballs as much as possible without moving his or her neck. The camera116 captures images including the cornea 302 of the user 300 when theuser 300 is gazing at the nine points including the points Q₁ to Q₉.

FIG. 6 is a schematic diagram illustrating the position coordinates ofthe cornea 302 of the user 300. The gaze detection unit 220 in the gazedetection device 200 analyzes the images captured by the camera 116 anddetects bright spots 105 derived from the infrared light. When the user300 gazes at each point by moving only his or her eyeballs, thepositions of the bright spots 105 are considered to be stationaryregardless of the point at which the user gazes. Thus, on the basis ofthe detected bright spots 105, the gaze detection unit 220 sets atwo-dimensional coordinate system 306 in the image captured by thecamera 116.

Further, the gaze detection unit 220 detects the center P of the cornea302 of the user 300 by analyzing the image captured by the camera 116.This is realized by using known image processing such as the Houghtransform or an edge extraction process. Accordingly, the gaze detectionunit 220 can acquire the coordinates of the center P of the cornea 302of the user 300 in the set two-dimensional coordinate system 306.

In FIG. 5, the coordinates of the points Q₁ to Q₉ in the two-dimensionalcoordinate system set for the display screen displayed by the imagedisplay element 108 are Q₁(x₁, y₁)^(T), Q₂(x₂, y₂)^(T), Q₉(x₉, y₉)^(T),respectively. The coordinates are, for example, a number of a pixellocated at a center of each point. Further, the center points P of thecornea 302 of the user 300 when the user 300 gazes at the points Q₁ toQ₉ are labeled P₁ to P₉. In this case, the coordinates of the points P1to P9 in the two-dimensional coordinate system 306 are P₁(X₁, Y₁)^(T),P₂(X₂, Y₂)^(T), . . . , P₉(X₉, Y₉)^(T). T represents a transposition ofa vector or a matrix.

A matrix M with a size of 2×2 is defined as Equation (1) below.

$\begin{matrix}{M = \begin{pmatrix}m_{11} & m_{12} \\m_{21} & m_{22}\end{pmatrix}} & (1)\end{matrix}$

In this case, if the matrix M satisfies Equation (2) below, the matrix Mis a matrix for projecting the gaze direction of the user 300 onto animage plane that is displayed by the image display element 108.

Q _(N) =MP _(N)(N=1, . . . ,9)  (2)

When Equation (2) is written specifically, Equation (3) below isobtained.

$\begin{matrix}{\begin{pmatrix}x_{1} & x_{2} & \ldots & x_{9} \\y_{1} & y_{2} & \ldots & y_{9}\end{pmatrix} = {\begin{pmatrix}m_{11} & m_{12} \\m_{21} & m_{22}\end{pmatrix}\begin{pmatrix}X_{1} & X_{2} & \ldots & X_{9} \\Y_{1} & Y_{2} & \ldots & Y_{9}\end{pmatrix}}} & (3)\end{matrix}$

By transforming Equation (3), Equation (4) below is obtained.

$\begin{matrix}{{\begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix} = {\begin{pmatrix}X_{1} & Y_{1} & 0 & 0 \\X_{2} & Y_{2} & 0 & 0 \\\vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & 0 & 0 \\0 & 0 & X_{1} & Y_{1} \\0 & 0 & X_{2} & Y_{2} \\\vdots & \vdots & \vdots & \vdots \\0 & 0 & X_{9} & Y_{9}\end{pmatrix}\begin{pmatrix}m_{11} \\m_{12} \\m_{21} \\m_{22}\end{pmatrix}}}{{Here},{y = \begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix}},{A = \begin{pmatrix}X_{1} & Y_{1} & 0 & 0 \\X_{2} & Y_{2} & 0 & 0 \\\vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & 0 & 0 \\0 & 0 & X_{1} & Y_{1} \\0 & 0 & X_{2} & Y_{2} \\\vdots & \vdots & \vdots & \vdots \\0 & 0 & X_{9} & Y_{9}\end{pmatrix}},{x = \begin{pmatrix}m_{11} \\m_{12} \\m_{21} \\m_{22}\end{pmatrix}}}} & (4)\end{matrix}$

If

Equation (5) below is obtained:

y=Ax  (5)

In Equation (5), elements of the vector y are known since these arecoordinates of the points Q₁ to Q₉ that are displayed on the imagedisplay element 108 by the gaze detection unit 220. Further, theelements of the matrix A can be acquired since the elements arecoordinates of a vertex P of the cornea 302 of the user 300. Thus, thegaze detection unit 220 can acquire the vector y and the matrix A. Avector x that is a vector in which elements of a transformation matrix Mare arranged is unknown. Since the vector y and matrix A are known, anissue of estimating the matrix M becomes an issue of obtaining theunknown vector x.

Equation (5) becomes the main issue to decide if the number of equations(that is, the number of points Q presented to the user 300 by the gazedetection unit 220 at the time of calibration) is larger than the numberof unknown numbers (that is, the number 4 of elements of the vector x).Since the number of equations is nine in the example illustrated inEquation (5), Equation (5) is the main issue to decide.

An error vector between the vector y and the vector Ax is defined asvector e. That is, e=y−Ax. In this case, a vector x_(opt) that isoptimal in the sense of minimizing the sum of squares of the elements ofthe vector e can be obtained from Equation (6) below.

x _(opt)=(A ^(T) A)⁻¹ AT _(y)  (6)

Here, “−1” indicates an inverse matrix.

The gaze detection unit 220 uses the elements of the obtained vectorx_(opt) to constitute the matrix M of Equation (1). Accordingly, usingthe coordinates of the vertex P of the cornea 302 of the user 300 andthe matrix M, the gaze detection unit 220 can estimate a point at whichthe right eye of the user 300 is gazing on the moving image displayed bythe image display element 108 according to Equation (2). Here, the gazedetection unit 220 further receives information on a distance betweenthe eye of the user and the image display element 108 from the headmounted display 100 and corrects an estimated coordinate value at whichthe user gazes according to the distance information. It should be notedthat a deviation in the estimation of a gaze position according to thedistance between the eye of the user and the image display element 108may be ignored as an error range. Accordingly, the gaze detection unit220 can calculate a right-eye gaze vector connecting the gaze point ofthe right eye on the image display element 108 and the vertex of thecornea of the right eye of the user. Similarly, the gaze detection unit220 can calculate a left-eye gaze vector connecting the gaze point ofthe left eye on the image display element 108 and the vertex of thecornea of the left eye of the user. It should be noted that it ispossible to specify the gaze point of the user on the two-dimensionalplane with the gaze vector of only one eye, and calculate depthdirection information of the gaze point of the user by obtaining thegaze vectors of both eyes. Thus, the gaze detection device 200 canspecify the gaze point of the user. It should be noted that the methodof specifying the gaze point shown herein is only an example, and a gazepoint of the user may be specified using a different method.

The above is the configuration of the video display system 1.

<Operation>

FIG. 7 is a flowchart illustrating an operation of the head mounteddisplay 100 in the video display system 1. FIG. 7 illustrates anoperation of the head mounted display 100 for one frame of video data,and the processing illustrated in FIG. 7 is repeatedly executed.

As illustrated in FIG. 7, first, the reception unit 110 of the headmounted display 100 receives video data from the gaze detection device200 (step S701). The reception unit 110 converts the received videosignal into a signal in an MIPI format and transfers the signal to thedisplay unit 102.

The display unit 102 starts a display of the received video signal inthe MIPI format (step S702) and outputs a vertical synchronizationsignal from the output terminal 106 to the control unit 120. The controlunit 120 receives the vertical synchronization signal with the inputterminal 122.

On the other hand, the input terminal 121 of the control unit 120receives an input of the viewing timing of the user 300 from the gazedetection unit 220 (step S703). The input terminal 121 transfers thereceived viewing timing to the irradiation delay control unit 123 andthe imaging delay control unit 124.

The irradiation delay control unit 123 determines the amount of delay inthe next frame on the basis of the transferred viewing timing. Inaddition, the imaging delay control unit 124 determines the amount ofdelay in the next frame on the basis of the transferred viewing timingand the timing of the vertical synchronization signal (step S704). Thatis, the imaging delay control unit 124 adds a time from start of videodisplay to the actual viewing by the user to the transferred verticalsynchronization signal on the basis of the viewing timing, subtracts atime required for actual imaging from the start of imaging, and adds atime corresponding to one cycle of the frame in the video data when asubtraction result is a negative value to generate the imaging timingsignal. It should be noted that, after periodicity for the timing of theirradiation and the imaging is established, the irradiation delaycontrol unit 123 or the imaging delay control unit 124 may fix a delaytime from the vertical synchronization signal and generate theirradiation timing signal or the imaging timing signal.

The control unit 120 transfers the irradiation timing signal generatedon the basis of the determined amount of delay to the irradiation unit135, and transfers the generated imaging timing signal to the imagingunit 140 (step S705).

The irradiation unit 135 turns on the left eye LED 103 a and the righteye LED 103 b at the timing indicated by the transferred irradiationtiming signal, and irradiates the eyes of the user 300 with nearinfrared light (step S706).

In addition, the imaging unit 140 starts imaging of the eyes of the userusing the left-eye imaging camera 116 a and the right-eye imaging camera116 b at the timing indicated by the transferred imaging timing signalto obtain a captured image (step S707).

The imaging unit 140 transfers the captured image to the output unit180, and the output unit 180 outputs the transferred captured image tothe gaze detection device 200 (step S708).

Accordingly, it is possible to obtain a captured image obtained byimaging the eyes of the user corresponding to one frame. In the gazedetection device 200 having received the captured image, the imageanalysis unit 221 performs analysis to specify a cornea position in theimage of the user 300, and the detection unit 222 receives an analysisresult of the image analysis unit 221 to specify a gaze point of theuser 300. The video generation unit 250 generates video data of the nextframe on the basis of the gaze point detected by the gaze detection unit220, and outputs the video data from the output terminal 240 to the headmounted display 100. Further, at the same time, the imaging control unit223 generates a viewing timing and transfers the viewing timing to thecontrol unit 120 of the head mounted display 100 in synchronization withthe output of the video data.

The above is the operation of the video display system 1.

Specific Example

An operation of the video display system 1 and effects thereof will bedescribed using one specific example of the signal herein.

FIG. 8 illustrates a timing chart when the imaging timing is notcontrolled, and FIG. 9 illustrates a timing chart when the imagingtiming is controlled. In both drawings, a horizontal axis is a timeaxis. Here, the passage of time is indicated in the order of ms.Further, in both drawings, a left side indicates which signal is in thevideo display system 1. In addition, here, it is assumed that a periodof a display of one frame of video is 11 ms. That is, in the videodisplay system 1, it is assumed that it is necessary for a process ofdisplaying one frame of the video, imaging the eyes of the user,specifying the gaze point of the user from the captured image, andgenerating data of one next frame from the specified gaze point to beperformed for 11 ms.

In FIGS. 8 and 9, each signal indicates a timing of the signal S1 outputfrom the output terminal 240, a timing at which the input terminal 111receives a signal, a timing of a signal S2 output from the outputterminal 113, a timing at which the input terminal 104 receives asignal, a timing of a signal S3 output from the output terminal 106, atiming at which the input terminal 122 receives a signal, a timing of asignal S4 output from the output terminal 126 of the control unit 120 (asignal defining an imaging start timing), a timing at which the imagingunit 140 receives a signal, a timing of a signal S5 output from theimaging unit 140, which is a timing at which the captured image isoutput, a timing at which the input terminal 182 receives the capturedimage, a timing of a signal S6 output from the output terminal 181,which is a timing at which the captured image is transmitted from thehead mounted display 100 to the gaze detection device 200, a timing atwhich the input terminal 210 receives the captured image, a timing of asignal S7 output from the input terminal 210 to the gaze detection unit220, a timing at which the image analysis unit 221 receives a signal, atiming of a signal S8 output from the image analysis unit 221, and atiming of a signal S1 output from the output terminal 240 in order fromtop.

As illustrated in FIG. 8 or 9, various processing delays occur in thevideo display system 1. In the case of FIG. 8, there are delays such asa delay of about 1 ms due to format conversion of the video signal inthe reception unit 110 (see the signals S1 and S2 in 0 ms to 1 ms inFIG. 8), a delay of about 1 ms caused between reception of the videodata in the display unit 102 and an output of the verticalsynchronization signal (see the signal S2 and the signal S3 in 1 ms to 2ms in FIG. 8), a processing delay of about 0.5 ms in the control unit120 (see the signal S3 and the signal S4 in 1 ms to 2 ms in FIG. 8), aprocessing delay of 0.5 ms until the captured image is output by imagingin the imaging unit 140 (see the signal S4 and the signal S5 before andafter 2 ms in FIG. 8), a processing delay in the output unit 180 (seethe signal S5 and the signal S6 in 2 ms to 3 ms in FIG. 8), a transferdelay of about 0.5 ms from the input terminal 210 to the gaze detectionunit 220 (the image analysis unit 221) (see the signal S6 and the signalS7 in 3 ms to 4 ms in FIG. 8), a processing delay of about 3 ms due tothe analysis in the image analysis unit 221 (see the signal S7 and 221in 3 ms to 7 ms in FIG. 8), and a processing delay of about 0.2 ms dueto an output process from the image analysis unit 221 (see 221 andsignal S8 in 6 ms to 7 ms in FIG. 8). As a result of these delays, thegaze point information is transferred to the video generation unit 250just before 7 ms in the example of FIG. 8.

Therefore, when the imaging timing control is not performed, the videogeneration unit 250 has time to prepare video data only for about 4 msbetween 7 ms and 11 ms in FIG. 8.

On the other hand, as shown in this present embodiment, it is assumedthat the imaging timing control is performed as shown in the timingchart of FIG. 9. That is, it is assumed that the control unit 120 (theimaging delay control unit 124) has performed the control to advance theoutput of the signal (S4) indicating the imaging start timing asindicated by the arrow in FIG. 9. Then, a timing at which the gaze pointis transferred to the video generation unit 250 can be naturallyadvanced.

Strictly speaking, the signal at 1 ms is used as a trigger, and anothersignal at a time earlier than that cannot be used as a trigger, andtherefore, the imaging start timing of the next frame is actuallydetermined. That is, the imaging delay control unit 124 determines theimaging timing (see the signal S4 at 9 ms in FIG. 9) at which the eyesof the user viewing the next frame are imaged according to the timing ofthe transferred vertical synchronization signal. The eyes of the uservisually recognizing the video data indicated by the signal S1 at 11 msare imaged on the basis of this imaging timing.

Therefore, by performing the imaging timing control, the information onthe gaze point is transferred to the video generation unit 250 earlierin FIG. 9. Therefore, the video generation unit 250 can generate thevideo data of the next frame over a period of 8 ms between 3 ms and 11ms, and room can be given for a process of generating the video data inthe video generation unit 250.

It should be noted that the amount of delay caused by the variousprocesses or transfers illustrated in FIGS. 8 and 9 is merely anexample, and it is apparent that it is necessary to change the imagingstart timing indicated by the imaging timing signal according to theamount of delay caused by the respective processes.

CONCLUSION

With the video display system according to the present embodiment, it ispossible to instruct start of imaging of the eyes of the user 300 in theimaging unit 140 on the basis of the timing at which the display unit102 receives new data and starts display thereof (performs verticalsynchronization) in advance. Therefore, it is possible to image the eyesof the user exactly at the timing at which the user 300 actually viewsthe displayed video after the start of the vertical synchronization.Accordingly, it is possible to advance the transfer timing of thecaptured image to the gaze detection unit 220 relative to the start ofimaging of the imaging unit 140 from the timing at which the displayunit 102 ends the display of the image, and therefore, room can be givenfor generation of the video data in the video generation unit 250.

<Supplements>

It is apparent that the video display system according to the aboveembodiment is not limited to the embodiment but may be realized byanother scheme. Hereinafter, various modification examples will bedescribed below.

(1) In the above embodiment, the wearable device 100 and the gazedetection device 200 are connected by the USB cable, but the presentinvention is not limited thereto, but a part or all (a part or all of apath of S1, a path of S6, and a path from the output terminal 230 to theinput terminal 121) may be replaced with wireless communication.Further, a cable for connecting the wearable device 100 to the gazedetection device 200 and transferring information may be realized by onecable instead of a plurality of cables as illustrated in FIG. 1.

(2) In the above embodiment, the control unit 120 receives the viewingtiming of the user from the gaze detection device 200 and determines theirradiation timing of the imaging unit 140 and the irradiation unit 135.However, the present invention is not limited thereto. The irradiationdelay control unit 123 may perform feedforward (feedback) for a fixedtime. Similarly, the imaging delay control unit 124 may performfeedforward for a fixed time. Further, alternatively, the control unit120 may store a plurality of fixed times with different time lengthsaccording to attributes of the user, determine, for example, a fixedtime length according to an age of the user as the attributes of theuser, and set the fixed time as a feedforward time. The fixed timelength may be set to be longer as the age of the user 300 is higher. Inaddition, a user registration function may be provided in the videodisplay system 1. In this case, it is possible to reduce a process ofthe imaging control unit 223 by storing the viewing timing of each userin advance.

(3) In the above embodiment, the gaze detection device 200 and thewearable device 100 are described as separate devices, but as describedabove, the gaze detection device 200 may be embedded in the wearabledevice 100. Further, in this case, only some of functions of the gazedetection device 200 may be included in the wearable device 100.

(4) In the above-described embodiment, the irradiation unit 135 may turnon the left eye LED 103 a and the right eye LED 103 b at all times.

(5) Although not shown in the above embodiment, the wearable device 100may include a storage unit for storing the video data, the viewingtiming, and the captured image captured by the imaging unit 140.Similarly, the gaze detection device 200 may also include a storage unitfor storing the received captured image and the like.

(6) In the above-described embodiment, a scheme for controlling thetiming for imaging the eyes of the user in the gaze detection device isrealized by each processor constituting the gaze detection deviceexecuting a given function, this may be realized by a logical circuit(hardware) formed of an integrated circuit (an integrated circuit (IC)chip or a large scale integration (LSI)), a field programmable gatearray (FPGA), or the like, a dedicated circuit. Further, the circuit maybe realized by one or a plurality of integrated circuits, or thefunctions of the plurality of functional units described above may berealized by one integrated circuit. The LSI may be called VLSI, superLSI, ultra LSI, or the like according to an integration difference. Inaddition, the control of the imaging timing and the control of theirradiation timing by the control unit 120 may be realized by softwareby the control unit 120 executing a timing control program (videodisplay program) for timing control.

Further, the timing control program may be recorded in aprocessor-readable recording medium, and as the recording medium, a“non-transitory tangible medium” such as a tape, a disk, a card, asemiconductor memory, or a programmable logic circuit can be used.Further, the timing control program may be supplied to the processor viaany transmission medium (a communication network, broadcast waves, orthe like) capable of transmitting the timing control program. In thepresent invention, the timing control program can also be realized inthe form of a data signal embedded in a carrier wave, which is embodiedthrough electronic transmission.

It should be noted that the timing control program may be installedusing, for example, a script language such as ActionScript or JavaScript(registered trademark), an object oriented programming language such asObjective-C, Java (registered trademark), C++, or C#, a markup languagesuch as HTML5, or the like.

(7) The respective configurations and respective supplements may beappropriately combined.

EXPLANATION OF REFERENCES

-   -   1 Video display system    -   100 Wearable device (head mounted device)    -   103 a Infrared light source (right eye LED)    -   103 b Infrared light source (left eye LED)    -   104 Input terminal    -   105 Bright spot    -   106 Output terminal    -   108 Image display element    -   111 Input terminal    -   112 Hot mirror    -   113 Output terminal    -   114, 114 a, 114 b Convex lens    -   116 Camera    -   116 a Right-eye imaging camera    -   116 b Left-eye imaging camera    -   120 Control unit    -   121 Input terminal    -   122 Input terminal    -   123 Irradiation delay control unit    -   124 Imaging delay control unit    -   125 Output terminal    -   126 Output terminal    -   130 Image display system    -   135 Irradiation unit    -   140 Imaging unit    -   150 Housing    -   152 a, 152 b Lens holding unit    -   160 Fitting harness    -   170 Headphone    -   180 Output unit    -   181 Output terminal    -   182 Input terminal    -   200 Gaze detection device    -   210 Input terminal    -   220 Gaze detection unit    -   221 Image analysis unit    -   222 Detection unit    -   223 Imaging control unit    -   230 Output terminal    -   240 Output terminal    -   250 Video generation unit

1. A video display system comprising: a wearable device including areception unit that receives a video, a display unit that displays thevideo received by the reception unit, an irradiation unit thatirradiates eyes of a user with near infrared light, and an imaging unitthat images the eyes of the user viewing the video displayed on thedisplay unit, on the basis of the near infrared light, the user wearingthe wearable device and viewing the video; a gaze detection unit thatdetects a gaze point of the user on the basis of a captured imagecaptured by the imaging unit; and a video generation unit that generatesa video to be displayed on the wearable device on the basis of the gazepoint detected by the gaze detection unit, wherein the wearable deviceincludes a control unit that instructs an imaging start timing to theimaging unit so that the imaging of the imaging unit can be executed ata timing at which it is estimated that the user is viewing the frameeach time each frame of the video to be displayed on the display unit isdisplayed.
 2. The video display system according to claim 1, wherein:the reception unit converts video data of the received video into aformat for displaying the video on the display unit, the display unitoutputs a synchronization signal indicating start of display of thevideo data converted by the reception unit, and the control unitoutputs, to the imaging unit, an instruction signal for instructing thestart timing to the imaging unit on the basis of the synchronizationsignal.
 3. The video display system according to claim 2, wherein thecontrol unit outputs the instruction signal according to a period oftime from start of display of the frame on the display unit to viewingby the user.
 4. The video display system according to claim 1, whereinthe control unit further controls a timing at which the irradiation unitirradiates the eyes of the user with the near-infrared light.
 5. A videodisplay method comprising: a reception step of receiving a video; adisplay step of displaying the video received in the reception step; anirradiation step of irradiating eyes of a user with near infrared light;a control step of instructing an imaging start timing so that imagingcan be executed at a timing at which it is estimated that the user isviewing the frame each time the frame of the video to be displayed inthe display step is displayed; an imaging step of imaging the eyes ofthe user viewing the video displayed in the display step, on the basisof the near infrared light according to the instructed start timing; agaze detection step of detecting a gaze point of the user on the basisof a captured image captured in the imaging step; and a video generationstep of generating a video to be displayed on the basis of the gazepoint detected in the gaze detection step.
 6. A video display programcausing a computer to execute: a reception function of receiving avideo; a display function of displaying the video received using thereception function; an irradiation function of irradiating eyes of auser with near infrared light; a control function of instructing animaging start timing so that imaging can be executed at a timing atwhich it is estimated that the user is viewing the frame each time theframe of the video to be displayed using the display function isdisplayed; an imaging function of imaging the eyes of the user viewingthe video displayed using the display function, on the basis of the nearinfrared light according to the instructed start timing; a gazedetection function of detecting a gaze point of the user on the basis ofa captured image captured using the imaging function; and a videogeneration function of generating a video to be displayed on the basisof the gaze point detected using the gaze detection step.